MDP solver for a class of location-based decisioning tasks

I applied an MDP approach to a class of stochastic planning tasks drawn from a location-based application scenario. The tasks involve generating the optimal visitation sequence across a number of geographically located “stores” at which one or more valuable resources are located, governed by expected utility based on travel times and acquisition rewards. I evaluated four factored MDP solvers based upon a dynamic bayesian network formulation. I created a composite MDP solver that drew from each of the four techniques. My composite MDP solver applies a symbolic heuristic approach as the main MDP solver, using the other three techniques for computing the heuristic. The approach exploits the strong reachability structure of the task to achieve reasonable compute times. I conclude that it is now feasible to attack moderately sized task instances from this class of problems. I discuss some practical lessons learned along the way in applying these algorithms and identify some new challenges.